Valid p-values and expectations of p-values revisited

Net risk reclassification p values: valid or misleading?

BACKGROUND The Net Reclassification Index (NRI) and its P value are used to make conclusions about improvements in prediction performance gained by adding a set of biomarkers to an existing risk prediction model. Although proposed only 5 years ago, the NRI has gained enormous traction in the risk prediction literature. Concerns have recently been raised about the statistical validity of the NRI...

P-values for classification

Let (X,Y ) be a random variable consisting of an observed feature vector X ∈ X and an unobserved class label Y ∈ {1, 2, . . . , L} with unknown joint distribution. In addition, let D be a training data set consisting of n completely observed independent copies of (X,Y ). Usual classification procedures provide point predictors (classifiers) Ŷ (X,D) of Y or estimate the conditional distribution ...

Understanding posterior p-values

Posterior predictive p-values do not in general have uniform distributions under the null hypothesis (except in the special case of pivotal test statistics) but instead tend to have distributions more concentrated near 0.5. From different perspectives, such nonuniform distributions have been portrayed as desirable (as reflecting an ability of vague prior distributions to nonetheless yield accur...

Very small P values.

Subsampling p-values

A construction of p-values for hypothesis tests based on subsampling and the related m out of n bootstrap is introduced. The p-values are based on a modification of the usual subsampling hypothesis tests that involves an appropriate centering of the subsampled or bootstrapped test statistics as in the construction of confidence intervals. This modification makes the hypothesis tests more powerf...